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Stimulating task interest: human partners or chatbots? 
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Abstract. The aim of this research project was to examine the impact of chatbots 
as conversation partners. First and second year Japanese university students 
(n=120) from a private university in Southwest Japan were randomly assigned to 
either conduct a speaking task with an Artificial Intelligence (AI) chatbot or human 
partner. Preliminary analysis of the data suggests that student interest in interacting 
with the chatbot conversation partners decreased across the experiment period, 
whilst interest in performing identical tasks with their peers (human partners) 
remained relatively stable. The findings suggest that educators and administrators 
should be cautious about relying entirely on AI conversation partners as a 
substitute to human partners if they wish to stimulate and maintain student interest 
levels in conversation tasks. Furthermore, teachers should carefully consider 
students’ language proficiency and communicative ability before designing and 
implementing speaking tasks that involve the use of AI conversation partners. 
Using chatbots as an extension of human-human conversation activity practice 
and not a replacement is recommended in order to maintain student interest and 


engagement across a language program. 
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1. Introduction 


This study is the first phase in the potential development and integration of using 
AI chatbots as conversation partners within a Japanese compulsory English 
language program. Through firstly identifying, understanding, and then exploiting 
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the benefits that chatbots offer students, educators and administrators may be better 
able to increase blended learning task-based opportunities for learners. 


1.1. Education and student interest 


Research over recent decades has linked student interest levels to vital components 
of learning, for example: attention, persistence, recall, and the quality of students’ 
learning outcomes (Hidi & Berndorff, 1998). Student interest has been shown to be 
connected to positive learning outcomes and, more importantly, also influence what 
learners choose to study in future years (Harackiewicz & Hulleman, 2010). Given 
the importance that interest plays in learning and its influence on positive academic 
and future outcomes, it is surprising that little research has been conducted on 
measuring student interest within the domain of language learning. Research into 
the development of interest has seen Hidi and Renninger (2006) propose a four- 
phase model of interest development. This model describes four phases in the 
development and strengthening of student interest. These phrases are triggered 
situational interest, maintained situational interest, emerging individual interest, 
and well-developed individual interest. It is important for both educators and 
administrators to be conscious of how student interest is triggered and maintained 
if we are to truly develop students’ long-term interest in English. 


1.2. Language learning and chatbots 


Today, more and more people are learning English through online technologies. 
With over 1.7 billion English language learners across the globe, only a small 
fraction of these learners have access to traditional language learning resources 
and formal classroom teaching (British Council, 2013). With the explosion of 
educational technology and the increased accessibility of mobile devices, education 
providers are actively seeking out ways to utilize chatbots as conversation partners. 
As Atwell (1999) noted, chatbots provide an opportunity for students to conduct 
conversational practice outside of the classroom. For an overview of the chatbot 
used in this study, Cleverbot.com, see Fryer and Carpenter (2006). 


Chatbots — also known as ‘conversational agents’ — are software applications that 
simulate human speech for the purposes of replicating a conversation or interaction 
with a human conversation partner. There are two main ways chatbots are offered to 
language learners. Firstly, through computer applications and also via online apps 
that can be accessed on mobile devices. This independent learning opportunity has 
gained interest within computer assisted language learning based research (Goda 
et al., 2014). 
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2. Method 


This research project had three main focuses: (1) to examine the suitability of 
chatbots as conversation partners; (2) to compare the levels of students’ interest 
in English language learning: chatbots versus human partners at task, course, and 
domain levels; and (3) to identify the merits/demerits of chatbots versus human 
conversation partners. 


2.1. Participants 


Japanese university students (first year n=91 and second year n=29 students) from 
a private university in Southwest Japan were requested to complete a spoken 
conversation task with a chatbot or human partner. Participants were non-English 
majors and came from five of the university’s seven faculties (Engineering, 
Management, International Studies, Fine Arts, and Economics), with a Common 
European Framework of Reference for languages (CEFR) basic user level ranging 
from Al to B2. Students were allocated into three English levels following a 
placement test conducted by the university administration. All participants were 
studying within a coordinated compulsory English as a foreign language program 
consisting of two 15-week semesters per academic year with one 90-minute 
Listening and Speaking class per week. 


2.2. Procedure 


Participants were randomly assigned the same English conversation task with a 
chatbot or human partner. The task-based conversation took students approximately 
10 minutes. Following the task, students self-reported their perceptions of the 
experience, identifying three advantages and/or disadvantages of speaking with 
a chatbot or human partner. The final stage of the data collection was an online 
interest survey. The online measurement was used to capture students’ level of 
interest throughout the course of research. For an overview of the development 
of the interest measurement, see Thompson et al. (2015). The experiment was 
conducted four times during one (15-week) academic semester. 


2.3. Instrumentation 


Following completion of the speaking task, students noted three advantages and/ 
or disadvantages relating the conversation partner experience (chatbot or human). 
From the third class of the second semester and each time students completed the 
task-based conversation component (four times) of this study, students’ interest in 
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the conversational task was measured. The survey items were: (Item 1) this activity 
was personally meaningful, (Item 2) I enjoyed learning English in this activity, 
(Item 3) J liked using English in this activity, (Item 4) this activity was interesting, 
and (Item 5) it was fun to review English in this activity. The items were measured 
on a six-point Likert scale. 


3. Preliminary findings 
Preliminary analysis of the quantitative data collected in this study indicates 
interest levels in English and conversation tasks with a human partner did show a 


slight increase across the experiment (see Table |, Table 2, and Table 3). 


Table 1. First year students: English interest 


Week Item Mean Standard Deviation 
3 3.85 1.29 
15 4.38 1.10 


Table 2. Second year students: English interest 


Week Item Mean Standard Deviation 
3 3.53 1.27 
15 4.09 1.20 


Table 3. First year students: human partner conversation activity interest 


Week Item Mean Standard Deviation 
5 4.37 1.00 
12 4.61 1.15 


However, there was a slight decline (see Table 4) performing the identical 
conversation task with a chatbot. This decline in interest may be linked to comments 
students noted in the qualitative data suggesting that students found the chatbot 
responses inappropriate and the interaction as being unnatural. 


Table 4. First year students: chatbot partner conversation activity interest 


Week Item Mean Standard Deviation 
7 4.18 1.11 
12 3.84 1.08 


These preliminary findings suggest that educators and administrators should be 
cautious about relying entirely on chatbot conversation partners as a substitute to 
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human partners if they wish to stimulate and maintain student interest in learning 
tasks and English more generally. 


4. Conclusion 


The aim of this research project was to examine the suitability of chatbots as 
conversation partners and to compare the levels of students’ interest in English 
language learning: chatbots versus human partners at task, course, and domain levels. 


The preliminary findings from this study suggest that educators and administrators 
should be cautious about relying entirely on chatbot conversation partners as a 
substitute to human partners if they wish to stimulate and maintain student interest 
in learning tasks and English more generally. As with any study conducted within 
one university, the external validity of the results awaits further tests both nationally 
and internationally. 
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